Working Paper: Assessing Correspondence between Experimental and Non-Experimental Results in Within-Study-Comparisons

Acknowledgements: This research was supported by a collaborative NSF grant #2015-0285-00. In within-study comparison (WSC) designs, treatment effects from a non-experimental design, such as an observational study or a regression-discontinuity design, are compared to results obtained from a well-designed randomized control trial (RCT) with the same target population. The goal of the WSC is to assess whether the non-experimental and experimental designs yield the same results in field settings, and the contexts and conditions under which non-experimental methods replicate benchmark RCT estimates. A common analytic challenge with WSCs, however, is identifying appropriate criteria for determining whether non-experimental and benchmark results replicate. This paper examines methods for assessing correspondence in benchmark RCT and non-experimental results in WSC designs. We examine measures for assessing correspondence in results, and the relative advantages and limitations of these approaches. We identify two classes of measures: conclusionand distance-based correspondence approaches. Conclusion-based measures indicate correspondence in results if a researcher or policy-maker draws the same conclusion from the experiment and non-experiment. Distance-based measures investigate whether the difference in estimates is small enough to claim correspondence between methods. We use a simulation study to examine the statistical properties of correspondence measures, and recommend a new approach that combines traditional significance testing and equivalence testing in the same framework. The paper concludes with practical advice on assessing and interpreting results in WSC contexts. EdPolicyWorks